Cryptographic Veriication of Test Coverage Claims

The market for software components is growing, driven on the \demand side" by the need for rapid deployment of highly functional products, and on the \supply side" by distributed object standards. As components and component vendors proliferate, there is naturally a growing concern about quality, and the eeectiveness of testing processes. White box testing, particularly the use of coverage criteria, is a widely used method for measuring the \thoroughness" of testing eeorts. High levels of test coverage are used as indicators of good quality control procedures. Software vendors who can demonstrate high levels of test coverage have a credible claim to high quality. However, verifying such claims involves knowledge of the source code, test cases, build procedures, etc. In applications where reliability and quality are critical, it would be desirable to verify test coverage claims without forcing vendors to give up valuable technical secrets. In this paper, we explore cryptographic techniques that can be used to verify such claims. Our techniques have certain limitations, which we discuss in this paper. However, vendors who have done the hard work of developing high levels of test coverage can use these techniques (at very low additional cost) to provide credible evidence of high coverage, while simultaneously reducing disclosure of intellectual property. Moreover, if such methods can be perfected and popularized, they can have an important \leveling" eeect on the software market place: small, relatively unknown software vendors with limited resources can provide credible evidence of high-quality processes, and thus compete with much larger corporations.